Method and system for reconstructing obstructed face portions for virtual reality environment

ABSTRACT

A method and a system for reconstructing obstructed face portions are provided herein. The method may include the following steps: obtaining off-line 3D data, being 3D data of a head of a person not wearing a face-obstructing object, being an object which obstructs a portion of the face of the person; obtaining in real time, real-time 3D data, being 3D data of said head, wherein said person wears said face-obstructing object; applying a 3D transformation to at least a portion of the off-line 3D data, based on the real-time 3D data, to yield reconstructed real time 3D data, being real-time 3D data related to the obstructed face portions; and merging the reconstructed real time 3D data into the real-time 3D data. The system may implement the aforementioned steps over a computer processor.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. application Ser.No. 16/562,893, filed Sep. 6, 2019, which is a continuation of U.S.application Ser. No. 15/501,160, filed Feb. 1, 2017, now U.S. Pat. No.10,445,863, which was the National Stage of International ApplicationNo. PCT/IL15/50801, filed Aug. 4, 2015, which claims the benefit of U.S.Provisional Application No. 62/032,678, filed Aug. 4, 2014, each ofwhich is incorporated by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to virtual reality environments,and more particularly, to such environments integrating real images ofthe virtual reality environment participants.

BACKGROUND

Prior to the background of the invention being set forth, it may behelpful to set forth definitions of certain terms that will be usedhereinafter.

The term ‘virtual reality’ (VR) as used herein is defined as acomputer-simulated environment that can simulate physical presence inplaces in the real world or imagined worlds. Virtual reality couldrecreate sensory experiences, including virtual taste, sight, smell,sound, touch, and the like.

The term ‘virtual reality headset’ as used herein is defined as a headmounted display (HMD) by which a 3D computer-simulated environment isprojected stereoscopically into the eyes of the user, with each eyereceiving a slightly different point of view of the computer-simulatedenvironment. One such virtual reality headset known in the art is OculusRift™. The virtual reality headset may be implemented as any type ofstereoscopic visor and may be held within a helmet-type structure.

The term ‘3D model’ as used herein is defined as the product of 3Dmodeling being the process of developing a mathematical representationof any three-dimensional surface of object (either inanimate or living).The model can also be physically created e.g., using 3D printing devicesor even manually. 3D models may represent a 3D object using a collectionof points in 3D space, connected by various geometric entities such astriangles, lines, curved surfaces, etc. Being a collection of data(points and other information).

The term ‘3D data’ as used herein is defined as any data structure, ofany form and kind, derived from 3D objects. 3D data may include, forexample, 3D models, 3D images but also less structured data types.

One challenge of 3D virtual reality today is to incorporate actual imageor data of the users (e.g., virtual reality environment participants)for example in the view point of each of the other users in a seamlessmanner. As some of the virtual reality systems currently availablerequire some form of virtual reality headsets within which the near eyedisplay is being integrated, at least a portion of the face is beingobstructed by the headset.

In some 3D virtual reality applications, it is desirable to capture inreal time the image of the user and merge the image into the view ofeach of the other users. In such applications, a helmet such as avirtual reality headset or a face obstructing object serving as a neareye display may cause loss of data, for example portions the user'sface, which undermines the altogether user experience.

FIG. 1 illustrates a virtual reality environment 100 according to theprior art, in which both persons (users) 106 and 108 are wearing a neareye display unit 102 and 104 respectively, which happen to obstruct atleast a portion of their faces. Their respective views (what they seethrough the displays) 110 and 112 are usually 3D in nature and so eachviewing image contains two viewpoints (not shown for simplicity). Theviews 110 and 112, presented to the users via their respective near eyedisplays include a computer-simulated environment not showing realobjects 103 and 105. Views 110 and 112 of computer-simulated environmentare adjusted in real-time responsive to the movements of the user. Views110 and 112 present images of their counter participant 106A and 108Awearing the near eye display 102A and 104A which obstruct most of theirfaces in views 112 and 110. The face obstruction is an undesirableoutcome and undermines the overall user experience.

SUMMARY

Some embodiments of the present invention overcome the aforementioneddisadvantages of the prior art by providing a method and a system forreconstructing obstructed face portions for virtual realityenvironments. The method may include the following steps: obtainingoff-line 3D data, being 3D data of a head of a person not wearing aface-obstructing object, being an object which obstructs a portion ofthe face of the person; obtaining in real time, real-time 3D data, being3D data of said head, wherein said person wears said face-obstructingobject; applying a 3D transformation to at least a portion of theoff-line 3D data, based on the real-time 3D data, to yield reconstructedreal time 3D data, being real-time 3D data related to the obstructedface portions; and merging the reconstructed real time 3D data into thereal-time 3D data. The system may implement the aforementioned stepsover a computer processor.

These, additional, and/or other aspects and/or advantages of theembodiments of the present invention are set forth in the detaileddescription which follows; possibly inferable from the detaileddescription; and/or learnable by practice of the embodiments of thepresent invention.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of embodiments of the invention and to showhow the same may be carried into effect, reference will now be made,purely by way of example, to the accompanying drawings in which likenumerals designate corresponding elements or sections throughout.

In the accompanying drawings:

FIG. 1 is a schematic block diagram illustrating a virtual realityenvironment according to the prior art;

FIG. 2 is a schematic block diagram illustrating a virtual realityenvironment according to embodiments of the present invention;

FIG. 3 is a schematic diagram illustrating a virtual reality environmentaccording to embodiments of the present invention;

FIG. 4 is a schematic diagram illustrating a system in accordance withembodiments to the present invention;

FIG. 5 is a schematic block diagram illustrating a system in accordancewith embodiments according to the present invention;

FIG. 6 is a schematic diagram illustrating a sequence of actions takenin accordance with embodiments of the present invention;

FIG. 7 is a schematic block diagram illustrating an aspect of a systemin accordance with embodiments according to the present invention; and

FIG. 8 is a flowchart illustrating a method according to someembodiments of the present invention.

DETAILED DESCRIPTION

With specific reference now to the drawings in detail, it is stressedthat the particulars shown are by way of example and for purposes ofillustrative discussion of the preferred embodiments of the presenttechnique only, and are presented in the cause of providing what isbelieved to be the most useful and readily understood description of theprinciples and conceptual aspects of the present technique. In thisregard, no attempt is made to show structural details of the presenttechnique in more detail than is necessary for a fundamentalunderstanding of the present technique, the description taken with thedrawings making apparent to those skilled in the art how the severalforms of the invention may be embodied in practice.

Before at least one embodiment of the present technique is explained indetail, it is to be understood that the invention is not limited in itsapplication to the details of construction and the arrangement of thecomponents set forth in the following description or illustrated in thedrawings. The present technique is applicable to other embodiments or ofbeing practiced or carried out in various ways. Also, it is to beunderstood that the phraseology and terminology employed herein is forthe purpose of description and should not be regarded as limiting.

Some embodiments of the present invention address the challenge ofreconstructing a facial portion of a virtual reality participant incases where a portion of the face is obstructed, usually by a 3D virtualreality headset such as Oculus Rift™ headset or Google Glass™.

Operating in the 3D virtual reality domain makes the solutionspecifically challenging in order to reconstruct a real-time 3D datasuch as 3D image of the virtual reality participant's face in a mannerseamless to the other viewer-participants.

FIG. 2 is a schematic block diagram illustrating a virtual realityenvironment according to embodiments of the present invention. A scene200 in a virtual reality system is depicted, in which both persons(participants) 206 and 208 are wearing a near eye display or a VRheadset 202 and 204 respectively, which happen to obstruct their faces.However, as opposed to FIG. 1, their respective views 210 and 212 whichas presented via their near eye displays 202 and 204 show their counterparticipant not wearing the near eye display (real objects 203 and 205are also not shown as the background of the real world is being replacedwith a computer simulated environment). This is enabled by replacingpart of the image that includes the obstructing object with asupplementary or modified face images that are being updated in realtime, based on actual movements and/or expressions and/or gestures ofthe participants as will be explained in detail below.

FIG. 3 is a schematic block diagram illustrating a virtual realityenvironment according to some embodiments of the present invention.Here, a third participant (not shown) is viewing two other participants308A and 306A who wear helmets such as VR headsets 304A and 302A in thereal world 300 which also contain real objects 303 and 305. In thevirtual reality view 300B presented to the third user, the two otherparticipants 308B and 306B are presented without the helmets as thesewere seamlessly replaced with reconstructed face images.

FIG. 4 is a schematic diagram illustrating an aspect in accordance withsome embodiments according to the present invention. A user helmet 404such as a VR headset is shown with a plurality of sensors 402 along itsinner side (attachable to the user's face). The sensors are configuredto sense the user's gestures such as the eyes (e.g., pupil of the eyeand eyelid movement) and a computer processor is configured to modify abase image of the face based on the ongoing gestures and present them inreal time (or with as little latency possible). By way of illustration,sensors 402 may sense movements of the facial skin that are indicativeof the user laughing or smiling. In response, the base image of his orher face will undergo a process of image processing applying the effectof the laugh or the smile such as stretching various portions. Thisprocess is being updated in real time. Consequently, other viewers willbe presented with the modified smiling face image of the user which willbe seamlessly inserted into the image of the user, instead of theobstructed portion of the face.

FIG. 5 is a schematic block diagram illustrating a system 500 inaccordance with embodiments according to the present invention. System500 may include: a 3D data capturing device 510 configured to capture a3D data (e.g., image) 512 of a scene containing at least one personwearing a face-obstructing object, wherein the capturing is taken from aview point of a specified user. System 500 may further include acomputer processor 520 configured to: obtain data 550 indicative of, forexample, face portions obstructed by the face-obstructing object,possibly from a face capturing module 530 which in turn receives thedata from sensors 532-534 or other sources such as networked databases.A possible source for 2D images of users faces may be social networkssuch as add Facebook™ or LinkedIn™ which may store profile images ofusers. These 2D images may be used for generating 3D data of the head orthe face of the user.

Computer processor 520 may be further configured to reconstruct 3D dataof the (e.g., an image) of the obstructed face portions based on theobtained data. Computer processor 520 may be further configured to mergethe reconstructed face image into respective location at the capturedimage of the scene. System 500 may further include a near eye display570 configured to present the merged image to the specified user,wherein the merged image is placed into a computer-simulated environmentadjustable based on the view point of the specified user.

According to some embodiments of the present invention, theface-obstructing object comprises one or more sensors located on aninner side of said face-obstructing object, wherein the sensors areconfigured to carry out the obtaining data indicative of face portionsobstructed by the face-obstructing object.

According to some embodiments of the present invention, the sensing bythe sensors may be in the form of image capturing and wherein saidobtained data is an image of portions of the obstructed face.

According to some embodiments of the present invention, the sensors maybe configured to sense facial gestures, and wherein the reconstructingof a face image is carried out by modeling sensed gestures to change theface relative to a base image of the obstructed face portions capturedpreviously.

According to some embodiments of the present invention, computerprocessor 520 may be configured to obtain the data indicative of faceportions obstructed by the face-obstructing object comprises obtaining abase image of the obstructed face portions, which has been previouslycaptured.

According to some embodiments of the present invention, the obtaining ofa base image of the obstructed face portions, by the processor iscarried out via searching on networked databases.

According to some embodiments of the present invention, thereconstructing of a face image of the obstructed face portions may bebased on an obtained data which is carried out by the processor bymonitoring at least one of: position, orientation, profile, and vitalsigns (e.g., heart beat rate, respiratory system indicators etc.) of theat least one person wearing the a face-obstructing object, and applyingthe obtained data onto a three dimensional model of the face of the atleast one person, to yield a modified face image.

According to some embodiments of the present invention, theface-obstructing object comprises a near eye display.

According to some embodiments of the present invention, the near eyedisplay is incorporated within a helmet.

According to some embodiments of the present invention, thecomputer-simulated environment may be a three dimensional scene commonto the specified user and the at least one person wearing theface-obstructing object.

FIG. 6 is a schematic diagram 600 illustrating a sequence of actionstaken in accordance with embodiments of the present invention. In afirst step, off-line 3D data (e.g., in the form of a 3D model) of thehead of a person 610 is being obtained, possibly but not exclusivelybased on a plurality of 2D images of the person 612, 614, and 616. 2Dimages of the person 612, 614, and 616 may be captured in an off-linesession prior to wearing the helmet or may be obtained from third partysources such as networked databases.

Alternatively, the off-line 3D data may be obtained using structuredlight technology, prior to wearing the virtual reality headgear.Specifically, by using structured light, such as disclosed in WIPO PCTApplication Publication No. WO2013088442, which is incorporated hereinby reference in its entirety, a depth map of the head or face of theuser may be generated from which the off-line 3D data may be retrieved.

Then, in a second step, real-time 3D data (e.g., model) of the user 620is obtained, while the person is wearing a face obstructing object 622(e.g., the virtual reality headset) the real-time image is beingconstantly updated. In a third step a 3D transformation may be appliedin a reconstruction module 640 to off-line 3D data 610 or a portion ofit 618 (which corresponds with the borders of the obstructed portions634 which may be segmented out from 3D model 632). The 3D transformationmay be based on real-time 3D data (model) 620 and more specifically,real-time parameters 644 from which the appearance of obstructedportions 634 may be estimated.

Real-time parameters 644 may include, for example: position andorientation of the user's face, or expressions, but also how tired theperson is, and what type of emotions are being experienced in real time.These real-time parameters 644 are all being used in order to estimateand reconstruct the face portions that are being obstructed by thevirtual reality headset

The product of the reconstruction module (or step) is a reconstructedreal-time data (or model) 650 which includes a sub portion of area thatis affected by non-obscured portions 652, usually the inner cheeksaffected by changes of gestures made to the outer cheeks. The appearanceof this area may be estimated based on real-time parameters 644 relatingto the non-obstructed portions. Another area is the area that is notaffected by non-obstructed portions 654 (e.g., the eyes). The appearanceof the eyes may be estimated based on another model, and using meta data642 from sources external to real-time 2D data 622 such as whichdirection the person is looking at, given that in the virtual realityenvironment the person is interacting with another person (whoseposition is also known). Finally, reconstructed real-time 3D data (ormodel) 650 may be merged into real-time 3D data (or model) possibly withthe obstructed portions segmented out 632, to yield a reconstructedreal-time 3D data (or model) of the head of the person, with theobstructed portions reconstructed. Model 660 may then be generated intoa 3D image to be presented to the other person(s) participating in thevirtual reality environment.

FIG. 7 is a schematic block diagram illustrating an aspect of a systemin accordance with embodiments according to the present invention.Specifically, a schematic illustration explaining how the obstructedface portion may be reconstructed is presented herein. System 700 mayinclude a network 710 which may be the Internet serving as a source forface images of participants. As indicated above, these base images 720are usually raw images which may be harvested from social networkwebsites such as Facebook™.

Concurrently, a modeler 730 being implemented by software may generate a3D model 732 of the face that need to be reconstructed. The model may bebased on several 2D views of the head of the person whose face imageneed to be reconstructed. Then, both the base image 720 of the faceretrieved form the third party source and the 3D model 732 are fed intoa computer processor 740 which processes them, together with real-timeparameters 750 relating to the actual position and orientation andperhaps facial gestures, of the head of the participant whose face imageneeds to be reconstructed. The output of computer processor 740 is areconstructed face image 760 in which the actual position, nearbyenvironment and orientation of the participant are used in order toadjust the base image and to apply on the model so that thereconstructed face image will be presented in the virtual reality viewas natural as possible. Examples for real time parameters may bedirection of view of the participant, his or her physical condition suchas how tired he or she is which may affect their eye positions andgeneral appearance of their face and the like, their current mood e.g.happy/angry/sad etc., their nearby environment e.g. at home or outsideon the beach in a hot weather. This appearance-affecting data may beused by the modeler to reconstruct and modify the base image, by methodsknown in the art (e.g., of texturing and digital animation).

In some other embodiments, facial expressions may be deduced by thecontext of the virtual reality environment such as what has occurred tothe user in the virtual world (e.g. in a game) and so the reconstructedface image can be dynamically adjusted based on the virtual worldactivities.

FIG. 8 is a high level flowchart illustrating a method 800 according tosome embodiments of the present invention. It is understood that method800 may be implemented by an architecture that is different from theaforementioned architecture of system 500. Method 800 may include:obtaining off-line 3D data, being 3D data of a head of a person notwearing a face-obstructing object, being an object which obstructs aportion of the face of the person 810; obtaining in real time, real-time3D data, being 3D data of said head, wherein said person wears saidface-obstructing object 820; applying a 3D transformation to at least aportion of the off-line 3D data, based on the real-time 3D data, toyield reconstructed real time 3D data, being real-time 3D data relatedto the obstructed face portions 830; and merging the reconstructed realtime 3D data into the real-time 3D data 840.

According to some embodiments of the present invention, the off-line 3Ddata may be an off-line 3D model and the real-time 3D data may be areal-time 3D model. A model can be any mathematical or real structureindicative of spatial characterization.

According to some embodiments of the present invention, the method mayfurther include the step of generating a merged real-time 3D image fromthe merged reconstructed real time 3D data and the real-time 3D data.

According to some embodiments of the present invention, the real-time 3Ddata may be super-positioned over a virtual reality (VR) environment.

According to some embodiments of the present invention, the real-time 3Ddata may be obtained by monitoring at least one of: position,orientation, profile, and vital signs of the person wearing theface-obstructing object.

According to some embodiments of the present invention, the method mayfurther include the step of applying the obtained real-time 3D data ontoa 3D model of the face of said person, to yield a modified face image.

According to some embodiments of the present invention, the 3Dtransformation may further include using data obtained fromnon-obstructed face portions of the person for estimating obstructedface portions that are affected by changes to the non-obstructed faceportions. Specifically, the estimating may be carried out based onsensing facial expression from the real-time 3D data.

According to some embodiments of the present invention, the 3Dtransformation may further include estimating an appearance ofobstructed portions that are unaffected by changes to the non-obstructedface portions (e.g., the eyes) and using the estimation in the 3Dtransformation. Specifically, the estimating may be carried out based onmeta-data relating to said person.

According to some embodiments of the present invention, the obstructedportions of the at least one person may be at least partiallyreconstructed based on occurrences in the VR environment. It isunderstood that this feature may also be implemented to thenon-obstructed portions of the face.

According to some embodiments of the present invention, the obtaining ofthe off-line 3D data may be carried out via an external database thatmay store 2D images received from a computer network. Alternativelyobtaining of the off-line 3D data may be carried out via a 3D sensor,prior to beginning a session of the VR environment.

In the above description, an embodiment is an example or implementationof the inventions. The various appearances of “one embodiment,” “anembodiment” or “some embodiments” do not necessarily all refer to thesame embodiments.

Although various features of the invention may be described in thecontext of a single embodiment, the features may also be providedseparately or in any suitable combination. Conversely, although theinvention may be described herein in the context of separate embodimentsfor clarity, the invention may also be implemented in a singleembodiment.

Reference in the specification to “some embodiments”, “an embodiment”,“one embodiment” or “other embodiments” means that a particular feature,structure, or characteristic described in connection with theembodiments is included in at least some embodiments, but notnecessarily all embodiments, of the inventions.

It is to be understood that the phraseology and terminology employedherein is not to be construed as limiting and are for descriptivepurpose only.

The principles and uses of the teachings of the present invention may bebetter understood with reference to the accompanying description,figures and examples.

It is to be understood that the details set forth herein do not construea limitation to an application of the invention.

Furthermore, it is to be understood that the invention can be carriedout or practiced in various ways and that the invention can beimplemented in embodiments other than the ones outlined in thedescription above.

It is to be understood that the terms “including”, “comprising”,“consisting” and grammatical variants thereof do not preclude theaddition of one or more components, features, steps, or integers orgroups thereof and that the terms are to be construed as specifyingcomponents, features, steps or integers.

If the specification or claims refer to “an additional” element, thatdoes not preclude there being more than one of the additional element.

It is to be understood that where the claims or specification refer to“a” or “an” element, such reference is not to be construed that there isonly one of that element.

It is to be understood that where the specification states that acomponent, feature, structure, or characteristic “may”, “might”, “can”or “could” be included, that particular component, feature, structure,or characteristic is not required to be included.

Where applicable, although state diagrams, flow diagrams or both may beused to describe embodiments, the invention is not limited to thosediagrams or to the corresponding descriptions. For example, flow neednot move through each illustrated box or state, or in exactly the sameorder as illustrated and described.

Methods of the present invention may be implemented by performing orcompleting manually, automatically, or a combination thereof, selectedsteps or tasks.

The descriptions, examples, methods and materials presented in theclaims and the specification are not to be construed as limiting butrather as illustrative only.

Meanings of technical and scientific terms used herein are to becommonly understood as by one of ordinary skill in the art to which theinvention belongs, unless otherwise defined.

The present invention may be implemented in the testing or practice withmethods and materials equivalent or similar to those described herein.

While the invention has been described with respect to a limited numberof embodiments, these should not be construed as limitations on thescope of the invention, but rather as exemplifications of some of thepreferred embodiments. Other possible variations, modifications, andapplications are also within the scope of the invention. Accordingly,the scope of the invention should not be limited by what has thus farbeen described, but by the appended claims and their legal equivalents.

What is claimed is:
 1. A method comprising: sensing, by a sensor on aninner side of a face-obstructing object, facial gestures of faceportions of a person obstructed by the face-obstructing object; andreconstructing a 3D face image of the obstructed face portions bymodeling the sensed facial gestures to change the obstructed faceportions relative to a previously captured base image of the obstructedface portions of the person.
 2. The method according to claim 1, furthercomprising presenting the reconstructed 3D face image to a viewing uservia a display.
 3. The method of claim 1, further comprising capturing animage of a scene containing the person wearing the face-obstructingobject.
 4. The method according to claim 1, wherein the base image ofthe obstructed face portions further comprises unobstructed faceportions of the person.
 5. The method according to claim 1, furthercomprising retrieving the base image from a social network.
 6. Themethod according to claim 5, further comprising modifying the base imagebased on an environmental condition affecting the person.
 7. The methodaccording to claim 1, wherein the reconstructing the 3D face imagecomprises monitoring at least one of: position, orientation, or profile;and applying the obtained data onto a three dimensional model of theface of the person to yield a modified face image.
 8. The methodaccording to claim 1, wherein the reconstructing the 3D face imagecomprises monitoring vital signs of the at least one person wearing theface-obstructing object, and applying the obtained data onto a threedimensional model of the face of the at least one person, to yield amodified face image.
 9. A system comprising: a computer processorconfigured to: obtain, from a sensor on an inner side of aface-obstructing object, sensed facial gestures of face portions of aperson obstructed by the face-obstructing object; and reconstruct a 3Dface image of the obstructed face portions by modeling the sensed facialgestures to change the obstructed face portions relative to a previouslycaptured base image of the obstructed face portions of the person. 10.The system according to claim 9, further comprising a display configuredto present the reconstructed 3D face image.
 11. The system according toclaim 9, further comprising an image capturing device configured tocapture an image of a scene containing the person wearing theface-obstructing object.
 12. The system according to claim 9, whereinthe base image of the obstructed face portions further comprisesunobstructed face portions of the person.
 13. The system according toclaim 9, wherein the computer processor is further configured toretrieve the base image from a social network.
 14. The system accordingto claim 13, wherein the computer processor is further configured tomodify the base image based on an environmental condition affecting theperson.
 15. A computer program product comprising a non-transitorycomputer-readable storage medium containing computer program code for:sensing, by a sensor on an inner side of a face-obstructing object,facial gestures of face portions of a person obstructed by theface-obstructing object; and reconstructing a 3D face image of theobstructed face portions by modeling the sensed facial gestures tochange the obstructed face portions relative to a previously capturedbase image of the obstructed face portions of the person.